NASA-CR-193161 


A Model-based Approach for Detection of Objects in Low Resolution 

Passive Millimeter Wave Images 


CO 

rH 

■4 

CO 

CM 

I 

ro 

o 

z: 


An interim report for NASA Grant NAG- 1-1371 
"Analysis of Image Sequences from Sensors for Restricted Visibility Operations" 
for the period January 24, 1992 to January 23, 1993 


/A/ oCo^c.’. 

/(£> * • / 

NASA Technical Officer: 

Dr. Randall L. Harris, Sr., 

Flight Management Division, Mail Stop 152E 
NASA Langley Research Center 
Hampton, Virginia 23681 



ro 

u 

c 

3 


00 

o 

CO 

SO 

fH 

o 


o 

N 

rci 

o 


CO 

O U 
UJ UJ 

00 -o 

< CO 
S3 O 

l 

J U. Ui 

UJ O > 

o 

O Z CO 
s: o co 

< 

<£ X 
O 

UJ Z 

«-s H- O 

rH UJ 
O Q 

ro oT -J 

O' o a 

U UJ 

| UJ 
esc X or: 
o a 

| <1 3E 

<c o o 

c/3 or. -J 

«£ IX 
Z X Z 

vy <C w 


rO 

O' 

O 


a. 


sO 

fM 


E 

■ c 

u to 

a) n 
+j e-* 

c fO • 

0 J > 


U) J 

LU 

O Cm 
C O 

X O' 


or 

UJ CM 

uj • 


c 

3 

0) 

ro 

5/) 

m 

c 

ra 

> 


•* >V 

4J Ifl 
V- C 

o c 

ex (u 

CP Ci 
0C •**' 


Principal Investigator: Professor Rangachar Kasturi 
Graduate Students: Yuan-Liang Tang and Sadashiva Devadiga 


Department of Electrical and Computer Engineering 
The Pennsylvania State University 
University Park, PA 16802 


PRECEDING PAGE BLANK WOT FILMED 


ii 


Abstract 


We describe a model-based vision system to assist the pilots in landing maneuvers under 
restricted visibility conditions, the system has been designed to analyze image sequences obtained 
from a Passive Millimeter Wave (PMMW) imaging system mounted on the aircraft to delineate 
runways/taxiways, buildings, and other objects on or near runways. PMMW sensors have good 
response in a foggy atmosphere, but their spatial resolution is very low. However, additional data 
such as airport model and approximate position and orientation of aircraft are available. We 
exploit these data to guide our model-based system to locate objects in the low resolution image 
and generate warning signals to alert the pilots. We also derive analytical expressions trom the 
accuracy of the camera position estimate obtained by detecting the position of known objects in 
the image. 
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I. Introduction 


Federal regulations specify the minimum visibility conditions under which airlines may take 
off and land. These minima are a function of the types of airplane and airport equipment. 
Therefore, there is a great deal of interest in imaging sensors which can see through fog and 
produce a real world display which, when combined with symbolic or pictorial guidance 
information, could provide the basis for a landing system with lower visual minimum capability 
than those presently being used [1], 

Since the energy attenuation in the visible spectrum due to fog is very large [2] (Fig.l), 
sensors are being designed to operate at lower frequencies (e.g. 94 GHz) where the attenuation is 
lower providing the ability to see through fog. NASA Langley Research Center, in cooperation 
with industry, is performing research on an on-board imaging system using a passive sensor 
operating at this frequency. Images from such sensors are of very low spatial resolution (Fig.2). 
However, additional supporting information in the form of knowledge about the airport and the 
position, orientation and velocity of aircraft is generally available. Thus a model-based image 
analysis approach is feasible to segment the image and to detect and track objects on the ground. 
Information extracted from such an analysis is useful to generate warning signals to the pilot of 
any potential hazard. This paper describes such a model-based technique, which makes use of a 
priori information about the geometric model of the airport and camera position and attitude data 
provided by the Global Positioning System (GPS) and other instruments. 

The geometric model of the airport contains positions of the runways/taxiways and 
buildings, the navigation instruments provide the position of the aircraft, and on-board 
instruments provide the orientation of the aircraft (yaw, pitch and roll). We use this information 
to define regions of interest in the image where important features such as runways/taxiways, the 
horizon, etc. are likely to be present. Edges corresponding to these features of interest are 
detected within these regions. After delineating regions representing runways/taxiways, we look 
for objects inside and outside these regions. 
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Fig. 1. Atmospheric effects on electromagnetic radiation [2]. 



The data from radio navigation instruments are known only upto a certain accuracy 
depending upon the type of radio navigation instruments. For example, GPS data is updated once 
every second and it is likely that a few such updates are missed making camera position data to be 
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a few hundreds feet off. On-board instrument data is generally useful to obtain more accurate 
camera position data than the GPS-based data. An alternative approach is to use the information 
about the location of detected objects in the images with known world coordinates (e.g. 
intersection of runways/taxiways, corners of buildings, etc.) to obtain an improved estimate ol the 
camera position. This requires an analytical study of the relationships among the camera 
parameters, the resolution of the images, and the distances between the aircraft and objects. 

In Section II we present a block diagram of the complete system. In Section III we 
describe the analytical model that establishes the relationship between the position, orientation and 
other physical parameters of the camera and the attributes of the captured images. This model is 
useful to calculate the accuracy of camera position estimation using image based features. In 
Section IV we present the method for defining the regions of interest in the image using the 
camera parameters and airport model. Section V includes image processing steps that are used to 
find regions corresponding to major features in the image and to detect objects in these regions. 
Experimental results are presented in Section VI. We conclude the paper with a summary and a 
brief description of future work. 
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II. System Description 

In this section, we describe the functions of various modules of the system shown in Fig. 3 
and the interaction between them. The input model of the airport contains positions of the 
runways/taxiways, and buildings. The model transformation module will take this model and the 
camera state information (position and orientation) as inputs to define the regions of interest in 
the image plane. 

The image processing algorithms in the feature detection module operates within these 
regions of interest to detect the edges of the runway, horizon, etc. in the image. An edge is fitted 
to the edge pixels if enough edge pixels are found within the regions of interest. The module 
outputs parameters which define major regions in the input image. 



GPS On-board Instrument 


Fig. 3. System block diagram. 
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The object detection module detects objects in the image using different thresholds for 
each region. For example, since detection of objects on the runway is extremely important, a 
lower threshold is used to flag every object even if the contrast is low whereas a higher threshold 
is used to detect objects which are outside the runway such as buildings, etc. Locations of 
detected objects with known world coordinates is useful to estimate camera state parameters. 

The motion estimation module uses dynamic scene analysis methods to estimate camera 
state parameters as well as to detect velocities of objects on the ground. The outputs from this 
module will be useful to detect potential collisions and generate warning signals as appropriate. 

The camera state estimation module integrates information obtained about the position 
and velocity of the aircraft from various sensors and modules and outputs necessary data to the 
model transformation module. 
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III. Accuracy of Camera State Estimation from Image-based Features 

As we need to use the camera state estimated from locating features of known objects in 
the image during the period when the GPS is not updated, it is necessary to know the accuracy of 
such estimated positions and the factors that decide the accuracy. Hence, an analytical model that 
establishes the relationship between the camera parameters and the attributes of captured images 
is necessary for guiding the image analysis system. Sensor positional parameters include range 
(distance from the aircraft to the runway threshold), cross range (distance from the aircraft to the 
runway center line), altitude, and pitch, roll and yaw angles. Sensor imaging attributes include the 
number of pixels in the image and the optical angular view measured in degrees. We derive the 
inter-relationships among these parameters. Using these relationships we calculate the accuracy 
of the estimate of camera position based on a minimum resolvable movement ot features by one 
pixel in the image. We obtain these accuracies for three different types of cameras (PMMW, 
FLIR, HDTV) at six ranges. 

A. Analysis: Throughout the analysis, for convenience, we assume that the sensor is located at 
the center of gravity of the airplane. Hence we can use the terms sensor position and aircraft 
position interchangeably. We also neglect the effect of curvature of the earth. The system of 
reference axis that forms the basis of system of notations used to describe the position of the 
sensor is shown in Fig.4. The figure shows an airplane with three mutually perpendicular axes — 
pitch, roll and yaw — passing through the center of gravity of the airplane. The image plane is 
assumed to be perpendicular to the rolling axis with its vertical and horizontal axes coinciding 
with the yawing and the pitching axis of the airplane, respectively. 

Fig. 5 shows an imaging situation during landing where the aircraft is at (X c , Y c , Z c ), with 
pitching angle 0, zero yaw and zero roll angle. Let a = 90° - 0. The field of view of the camera is 
determined by two viewing angles: Aa defined in the same plane as 0 and Ap at right angles to Aa 
(Aa determines the vertical extent of the image and AP its horizontal extent). Even though the 
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image obtained by the sensor is always a rectangle, the ground area captured by the sensor is a 
trapezoid ABCD whose side length and area depends on Aa, Ap and various other sensor 
parameters like position, orientation etc. Note that a pixel in the image plane corresponds to a 
patch on the ground plane. We refer to this as a pixel-patch (see Fig.6). 



Z 



World coordinate system 


Fig. 4. Airplane-body axis (reproduced from "Airplane Aerodynamics" 
by Dommasch and Danieol Otto, 1967). 


Consider a point feature which has been detected at some pixel (p, q). Let the actual 
world coordinates of this feature be (P, Q, 0). since a pixel represents a patch on the ground, the 
camera could change in its position by certain amount while still retaining the image of the feature 
at the same pixel (p, q). Hence a camera pose estimation by passive triangulation will always give 
the same camera pose for nearby camera positions unless the change in camera position is large 
enough for the feature to be observed in the neighboring pixel. We define this minimum change in 
camera displacement as the sensitivity of the camera. Note that this is a measure of accuracy of 
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camera position estimate and is a function of the camera, image size in number of pixels, angular 
resolution, and the pixel location (p, q) in the image plane. 



Fig. 5. Image obtained by the sensor is projected towards the ground. 
Shaded area is the ground area covered by the sensor. 


Let N x and N y represent the number of pixels in the vertical and horizontal directions, 
respectively. The pixels are numbered -N x , .... 0 , ..., N x /2-l in the vertical direction and -Ny, 0, 
.... Ny/2-1 in the horizontal direction. The rolling axis of the plane is assumed to pass through the 
bottom right comer of the patch on the ground plane which corresponds to the center pixel in the 
image plane. Other pixels are referenced in a similar manner. The coordinates of the reference 
comer of the ground area covered by a pixel (p, q) can be estimated by the following relations. 


Aa 

X = X c + Z c tan(a + p — ), 

N v 


f = t c + 


Z c , Aft 

— r — tan(g — 1 -). 

. Aa. w M 
cos(a + p — ) y 

N Y 


(1) 
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For a non zero rolling angle (j), the ground coordinates (X\ Y') which corresponds to a pixel (p, q) 
in the image plane are obtained by replacing (p, q) in the above equation by (p\ q'), where 


p' = p cost })- q sin 0, and 
q' = psin0 + gcos (j). 


( 2 ) 


Since a pixel-patch is referenced by its bottom right corner of the pixel, the other three corners 
become the reference of its three neighboring pixels-patch as shown in Fig.7. Thus, the four 
comers of this pixel-patch (X^Yj ), i - 1, 2, 3, 4 , are obtained by using Eq. (1), where (p, q) are 
replaced by (ppqi), where 

Pl = pi cos 0 - qi sin 0, 

i 

qi - pi sin 0 + qi cos 0, 


( 3 ) 



Fig. 6. Ground area covered by the sensor. 
Each small trapezoid corresponds to 
a pixel in the actual image. 



Fig. 7. A pixel (p, q) projected 
towards the ground. 


Eq. (1) explicitly gives the relationship between the camera parameters (X c> Y c , Z c ), 0, 0, 
and a ground point corresponding to a pixel (p, q). We are now interested in computing the 
sensitivity of the imagery sensor. This is defined as the minimum change in a camera parameter 
that would move a fixed ground point to the next pixel in the image plane. We obtain this by 

! t 

taking the partial derivative of X\ and Fj with respect to the corresponding parameters. For 
example, 
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This derivation is an approximation to the amount of change in x[ for unit change in X c . 
Thus we estimate that the amount of change in X c in order to change X { to X 2 , or Y x to Y 4 (which 
define the comers of adjacent pixels) as 


Note that =°°, as expected. Sensitivity with reference to other parameters is defined in a 


similar manner. These are summarized in Table I. 


Sensor sensitivity is a function of various sensor parameters and sensor attitudes. Since 
the sensor plane is inclined to the ground plane, the sensitivity varies in the vertical and horizontal 
direction along the sensor plane and hence is a function of pixel number (p, q). Equivalently, the 
accuracy of estimation of sensor position using ground truth data is a function of pixel position as 
well as other parameters. For a given range, the estimation using features that are observed at the 
top half of the sensor are less accurate because of the large ground area represented by these 
pixels. Also for a given p, the accuracy decreases as we move towards the border of the sensor in 
the horizontal direction. In summary, the accuracy of estimation is a function of sensor 
characteristic and the ratio of the sensor view angle to the number of pixels in the image. 


B. Quantitative Results and Discussions: The sensitivity analysis described in the previous 

section was applied to three different sensors at six different positions (Table II). Sensitivities 
Sx c , Sy c , and Sz at the aim point (i.e., p=0, q-0) for various sensor positions are plotted in 

Y x 

Figs. 8, 9, and 10, respectively. Note that Sz is larger than Sz c at (0, 0) and hence a feature 

V 

would move to the next horizontal pixel before it moves to the next vertical pixel. Thus only Sz c 
is important. 

As expected, the sensitivity is the best for the sensor with the highest pixel resolution. 
Sensitivity also improves as the sensor is moved closer to the ground. It becomes poor for the 
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features that are located at the far end of the vertical axis (top of the sensor), i.e., for the objects 
that are located at the far end of the runway. Thus, as expected, the position and velocity of the 
aircraft can be computed to a setter accuracy by knowing the position of stationary objects on the 
ground that are closer to the aircraft. 


Table I. Sensor positional sensitivity equations. 


x r 

S* 

2 Z c sin(cos<|) • Aa / N x ) 

2 Z c sin(Aa / N x ) 


A c 

*i 

cos(2a + Aa / N x (2p + 1) cos<|) - 2q sin <j>) + 1 

oo 

cos(2a + Aa / N x ) + 1 

oo 

Yr 

% 

oo 

oo 



{Z c tan(g 4 Ap/ V v )/cos(a + p 4 Aa/ N x )}~ 
{Z c ton(q x A$/ Ny)/cos(a + PiAOL/ N x )} 

2Z C sm(Af}/ N y ) 
cos a cos(Ap / N y ) + 1 

Z r 

s# 

Sx c / tan(a + p x Aa / N x ) 

2 Z c sin(Aa / N x ) / sin(2a / N x ) 


cr 

Sy r cos(a + p x Aa / N x ) / tan(<ft Ap / N y ) 

OO 

0 

sg 

Sx c cos 2 (a + pi Aa / N x )l Z c 

sin( Aa / N x )l {cos a / cos(a + Aa / N x )} 



Sy c cos 2 (a + pi Aa / N x ) 

Z c tanC^Ap/ V^sinCa + ^Aa/V,.) 

oo 

4) 

s x 

2 Z c sin( cos <j> ■ Aa / N x ) 
cos(2a + Aa / N x (2 p + l)cos<|)- 2<?sin (J>) + 1 

oo 


S Y 

/ Zc(ASB§<\> + BSA/b<b) 

oo 


8A /§<}> = tania + ^Aa/ yV y )(-/7 sin cosmic Aa/ N x )cos(a + p x Aa/ N x ); a -90° +0; 

(p,q ] ) = (p,q)-, (p 4 ,q 4 ) = (p,q + l)‘ p x = p x cos <> — sin 0; p 4 = p 4 cos <}> - q 4 sin <f>; q x = p x siiKjH-^ cos <J>); 
q 4 = p 4 sin <J) -f- cos <)) ; 


SPP: Sensor Positional Parameters 
(X C ,Y C ,Z C ) Sensor position 

0 Pitch angle 

<)> Roll angle 


Field of view 


Sensor Characteristics 
Vertical Horizontal 
Aa Ap 


Number of pixels 


Sensitivity: Minimum change in the sensor positional parameters (X c , Y c , Z c , 0 , (j) ) that will make the object to 
appear in the next pixel either in the vertical (X: hence called as sensitivity in jt direction) or in the horizontal ( Y : 
hence called as sensitivity in y direction) direction of the sensor plane. Sj : Sensitivity in the direction 'j' due to 
the sensor positional parameter T computed at pixel (p, q) in the image plane. 


Table II. 
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Sensor Characteristic 


Sensor type 


Pixel 

(HxV) 




Field of view 
(HxV) deg. 




Sensor Positions 

(pitch = -3.0°, Roll = 0°, Cross range = 0 ft.) 
Location Range in ft. Altitude in ft. 



Threshold 

o 

o 

CAT II-DH 

908.1 

CAT I-DH 

2816.2 : 

Middle Marker 

4500.0 : 

1000' Altitude 

18081.1 

Outer Marker 

29040.0 



Sensitivity in direction of Range 


Sensitivity in the direction of Cross Range 


Resolution (degrees/pixel) 

Fig. 8. 


Resolution (degrees/pixel) 

Fig. 9. 


Sensitivity in the direction of Altitude 


0.10 0.15 0.20 0.25 0.30 0.35 

Resolution (degrees/pixel) 
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The results indicate that the accuracy of camera state estimation would be no better than 
the GPS data unless a high resolution sensor is employed. Note that these results do not consider 
potential improvements that can be obtained by motion stereo techniques using a large number of 
image frames. We are presently investigating the possibility of improving the accuracy of the 
computed sensor positional parameters by extending our analysis using this method. 
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IV. Model Transformation 

As noted earlier, the PMMW images are low contrast-low resolution images. Simple edge 
detection techniques on these images generate many noisy edge pixels in addition to those 
belonging to the true edges such as runways, sky etc. This problem is alleviated by defining 
regions of interest on the ground plane for each feature in the model and to perform 3D to 2D 
transformation. It also defines a region in the image plane where the horizon line should occur. 

A. Defining Regions of Interest for Runway Edges: The error in the expected location of a 
feature and its actual position in the image depends on several factors, most notably the accuracy 
of the camera position parameters used by the model transformation module. Furthermore, it is 
evident from our earlier analysis (Fig. 6) that the ground area covered by a pixel is a function of 
the position of the pixel in the image. Thus it is not reasonable to define the search space for each 
feature as a fixed number of pixels centered around the expected location in the image plane. 
Hence we define the region of interest in the 3D space and then apply transformation to get the 
corresponding region of interest in the image. The extent of the search space in the 3D space is 
determined by the estimated error in camera positional parameters (which are based on GPS and 
on-board instrument data). 

The geometric model of the airport contains a sequence of 3D coordinates of the vertices 
of the run way/taxi ways, which forms a polygon with n vertices: 

runway = { PJ , i = 1, 2 , ..., n, 

where P t = (X if Y it Z ;) T is one of the vertices of the polygon. Note that Zi — 0- ^i^i+1 specifies 
an edge of the polygon. The region of interest is defined as a rectangle on the ground which 
encloses the edge. Therefore, each edge PiPi+j of the polygon is associated with the region of 
interest defined by four points b( = (Xp Yp Zj), j = 1, ..., 4, and Zj = Z f . 

The width of the region of interest is defined as a function of the width of the 
runway/taxiways, w, accuracy of the GPS data, g (0 < g < 7), and the accuracy of the on-board 
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instrument, d (0 < d < 1). Note that g and d are determined by the specification and 
characteristics of these instruments. This relationship is given by 


width(w,g,d) = — . (6) 

gd 

Note that the minimum width is 0.2w when g-d-1 , which corresponds to ± 10% potential 
displacement of runway edge feature. To limit the search area from being a large traction of the 
runway width we limit the search width to 0.4w even if gd<0.5. 

After defining the region of interest for each edge, 3D to 2D coordinate transformation is 


where 


Xp 


~X~ 

Xq 

= PRT 

Y 

Xr 


Z 

_X_ 


_1_ 





'1 0 

0 


0 1 

0 

P = 

0 0 
i 0 
./ 

1 


0 


- sin y cos 


R = 


~COS\J/COS0 

cos ij / sin 0 sin 0 — sin \j/ cos (J> sin y sin 0 sin $ + cos y sin <f) 
cos y sin 0 cos <|) + sin y sin <j) sin y sin 0 cos <}> 


0 


T = 


0 

'10 0 - x c 

0 1 0 -Y c 
0 0 1 -Z r 


0 0 0 1 


(7) 


( 8 ) 


-sin0 

-cos0sin(j) 

-COS0COS({) 

0 


, and 


(9) 


(10) 


are the perspective projection, rotation and translation transformation matrices, respectively, and/ 
is the focal length. After perspective projection, we need to consider the following special cases: 
A. the region of interest degenerates to a line in the image plane because the region is too 


far from the camera. 
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B. the region of interest in the image plane becomes very large because the edge is very 
close to the camera. 

For case A, a minimum width in the image plane is assigned in order to provide some search space 
for the feature detector. For case B, a maximum width in image space is defined to further 
restrict the region. In our experiment, for the aforementioned extreme cases, the minimum and 
maximum width of a region of interest are set to be 10 and 20 pixels, respectively. 

B. Defining Search Space for Horizon Line: When the vertical angular field of view is larger 
than 29, then a horizon line appears in the image (Fig. 11). The horizon is an important clue in 
estimating the camera orientation since it gives the roll angle information direcdy. Search space in 
the image plane is defined to locate this line. 



image plane 

Fig. 11. Horizon line in the image. 


Without loss of generality, consider the situation when the aircraft is heading towards the 
X axis of the world coordinate system. Assume the camera is located at point D (see Fig. 11) with 
pitch angle 0, and zero yaw and roll angles. Point A and B are on the top and bottom edge of the 
image, respectively. The horizon will then appear horizontally in the image plane as shown. The 
distance between this line and the center line of the image is given by HC = / tan 0. Since in the 
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above analysis roll angle has been assumed to be zero, the horizon appears parallel to the 
horizontal axis of the image plane. For any non zero roll angle, a simple roll transformation on 
this line will give the horizon in the image. The associated region of interest is defined to be 10 
pixels centered around the expected horizontal line in the image. 

It is possible for the projection of the region of interest onto the image plane to be partially 
outside the image boundary. In such cases, wee need to clip these regions so that the search 
space always remains within the confines of the image. This is done using the polygon clip and 
fill algorithm [4]. The regions of interest for both the runway and the horizon of the image 
sequence used in these experiment are shown in Fig. 12. 



Fig. 12. Regions of interest. 
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V. Runway Localization and Object Detection 

A. Runway Localization: In this part, we search for the expected features within the region of 
interest defined by previous module. This will significantly reduce the search time and also avoid 
spurious responses which are likely in such a low resolution input image. An accurate localization 
of the feature is necessary for estimation of motion parameters and camera pose. 

A Sobel edge detector is applied to the sensor image. We then select one of the four 
scanning directions (-45°, 0°, 45°, 90°) which is approximately orthogonal to the direction of the 
expected edge. Along each scan line we locate pixels with greatest edge strength. As the runway 
edge is supposed to be a straight line we fit a best line to these pixels. We also associate a 
measure of confidence for these detected edges based on the number of edge pixels detected 
along the line. 

B. Object Detection: In this part, the region inside and outside the runway/taxiways are 

separately checked for the existence of any stationary or moving objects. The image has three 
homogeneous regions, namely the sky, the runway/taxiways and the region outside the 
mnway/ taxi ways. Any objects on or outside the runway/taxiways are expected to have some 
deviation in graylevel from their respective homogeneous background. Hence, we use histogram- 
based thresholding for object detection. The thresholds which determine this deviation are set to 
be different for different regions. 

We generate a mask image which represents three homogeneous regions. Using this mask 
image, we generate the histogram and compute its standard deviation for each region separately 
(except for the sky region). The threshold value is determined as a function of the mean and the 
standard deviation, and any area which has graylevel lower than the threshold is considered as 
object regions. An object is assumed to have a reasonable size. This size restriction on the object 
can be used to ignore spurious responses resulting from the thresholding. Each object is then 
labeled based on 4-connectivity. 


! 
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VI. Experimental Results 

We have tested our algorithm on a test image provided by the TRW. This image was 
obtained using a single pixel camera located at a fixed point in space (a camera with an array of 
pixels is under development). The camera was mechanically scanned to obtain a 50x150 pixel 
image. This is the image shown in Fig.2. We were also provided with the model of the runway 
giving the 3D world coordinates of the runway comers, locations of the buildings etc. Using 
these data and the single image, we created a sequence of 30 frames to simulate the images from a 
moving camera. Frames 1 (original), 5, 10, and 15 from this sequence are shown in Fig. 13(a). 
Edge enhanced images corresponding to these frames are shown in Fig. 13(b). The regions ol 
interest defined on these frames are shown in Fig. 13(c). Delineated features superimposed on the 
images are shown in Fig. 13(d). Although all the edges are detected accurately in the example, it 
is likely that one or more edges of a polygon are not detected. To handle such situations we 
associate a degree of importance for each edge. For example, runway edges which are closer to 
the camera must be detected in the image whereas those corresponding to the far end of the 
mnway are usually very short and may or may not be detected. And overall confidence measure 
is associated with each detected region. 

Objects detected on the mnway in Frame 1 and those outside the mnway are shown in 
Fig. 14. Warning signals are generated for each object on or near mnway. Algorithms to track 
these in successive frames and estimate camera state using motion stereo are under development. 
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VII. Future Work and Conclusions 

In this paper, we have described a vision-based system to assist pilots during landing under 
restricted visibility conditions. The images obtained by a passive sensor is processed to detect 
major regions such as runways and objects inside and outside these regions. The image resolution 
is very poor, however, additional information in the form of airport geometric model, and camera 
position parameters are available to guide the segmentation algorithms. Objects are detected in 
each of these regions using thresholds computed separately for each region. Our results show 
that the model-based feature detection approach is quite accurate and the homogeneity 
assumption on regions for object detection is reasonable. The success of this model-based 
approach clearly depends upon the accuracy of the camera position parameters used to define 
search regions in the image. One of the methods for updating camera position information is 
triangulation using known objects. We have derived the accuracy of such an update as a function 
of camera characteristics and image parameters. 

At this stage, our system is able to detect the run way/ taxi ways and the objects inside and 
outside the runway/taxiways in each frame and to report their positions in the image. Since we 
have a moving camera, moving object situation, even the stationary objects appear to be moving 
in the image. Work is in progress to estimate the egomotion of the camera, to distinguish moving 
objects from stationary ones and to estimate the velocities of the moving objects. There is also 
potential to obtain more accurate camera state estimation using motion stereo from image 
sequences compared to using GPS data alone. 
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